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IMAGE CAPTURING APPARATUS AND METHOD THEREFOR 

BACKG ROUND OF THE INVENT ION 
FIE LD OF THE INVENTION 

This invention relates to an image capturing apparatus and a 
method therefor, and more particularly, is suitably applied, for 
example , to a video camera. 

DES CRI PT ION OF TH E RE LATED ART 

Conventionally, there is a video camera which composes a 
plurality of images captured with different amounts of exposure to 
produce an image having a wide dynamic range (i.e., an image which 
represents from a light region to a dark region). 

A method of capturing a plurality of images with different 
amounts of exposure as mentioned may employ, as an imager device, 
a solid-state imager device (CCD: Charge Coupled Device), and 
involves adjusting an electronic shutter, which is a function 
possessed by the CCD, to change the exposure time to capture a 
plurality of images in a time division manner* In this method, a 
charge is accumulated and read in an arbitrary field period in a 
manner similar to a normal imaging operation, and subsequently a 
charge is again accumulated and read utilizing a vertical blanking 
period to capture two different images with different exposure 
times within one field period. 



Also, as illustrated in Fig. 1, another method utilizes 
Neutral Density (ND) filters 3 having different transmittivities 
placed on respective pixels of an imager device 1 to capture an 
image which is spatially divided into segments having different 
amounts of exposure. Further, as illustrated in Fig. 2, another 
method involves providing a plurality of imager devices 5A, 5B, 
and capturing a plurality of images without reducing the spatial 
resolution by disposing ND filters in front of the respective 
imager devices, for example, placing an ND filter 7 in front of 
the imager device 5B, to give different transmittivities. 

The method of capturing a plurality of images having 
different amounts of exposure in a time division manner is capable 
of producing an image of higher resolution as compared with the 
method of capturing a plurality of images having different 
exposure amounts in a spatial division manner, and is simpler in 
structure and lower in cost as compared with the method of 
capturing a plurality of images having different amounts of 
exposure using a plurality of imager devices. It can therefore be 
said that this image capturing method is better than the other two. 

However, since the method of capturing a plurality of images 
having different amounts of exposure in a time division manner 
captures respective images having different amounts of exposure at 
different times, misregistration between the respective images, if 
caused by unintended movements of hands or the like, would result 
in a problem of a degraded image quality on a resulting composite 
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image produced by composing the captured images having different 
amounts of exposure due to the misregistration on the composite 
image . 

Referring now to Figs. 3A and 3B, explanation will be given 
of how such misregistration between respective images causes a 
degraded image quality of a composite image. Fig. 3 A one- 
dimensionally shows an image captured with long time exposure; and 
Fig. 3B one-dimensional ly shows an image captured with short time 
exposure, where misregistration equal to a length D occurs between 
the two images. When these two images are composed , disturbed 
image portions will be produced near the edges, as a composite 
image shown in Fig. 3C. 

SUMMARY OF THE INVENTION 

In view of the foregoing, an object of this invention is to 
provide an image capturing apparatus and a method therefor which 
are capable of producing a natural composite image corresponding 
to an object to be imaged. 

The foregoing object and other objects of the invention have 
been achieved by the provision of an image capturing apparatus for 
capturing a plurality of images with different amounts of exposure 
to compose the images into a single composite image, comprising 
misregistration amount detecting means for detecting the amounts 
of misregistration of respective images except for a reference 
image, wherein the reference image is selected from the plurality 
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of images, with respect to the reference image, misregistration 
correcting means for correcting misregistration of the respective 
images except for the reference image with respect to the 
reference image based on the misregistration amounts, and image 
composing means for composing the reference image and all of the 
respective misregistration corrected images except for the 
reference images . 

A misregistration amount of each of the images except for the 
reference image is detected with respect to the reference image, 
misregistration of each of the images except for the reference 
image with respect to the reference image is corrected on the 
basis of the misregistration amount, and then the reference image 
and the respective misregistration corrected images except for the 
reference image are composed, thereby making it possible to avoid 
a degraded image quality which would otherwise be produced when a 
plurality of images are composed without correction. 

The nature, principle and utility of the invention will 
become more apparent from the following detailed description when 
read in conjunction with the accompanying drawings in which like 
parts are designated by like reference numerals or characters. 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the accompanying drawings: 

Fig. 1 is a schematic diagram used for explaining exposure 
amount control based on spatial division; 
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Fig. 2 is a schematic diagram used for explaining exposure 
amount control using a multi-plate imager device; 

Figs. 3 A, 3B and 3C are schematic diagrams used for 
explaining how misregistration occurs; 

Fig. 4 is a block diagram illustrating the configuration of a 
video camera according to a first embodiment of the present 
invention; 

Fig. 5 is a schematic diagram used for explaining how the 
amount of exposure is controlled in a time division manner; 

Fig. 6 is a schematic diagram used for explaining 
misregistration between images ; 

Fig. 7 is a block diagram illustrating the configuration of 
misregistration amount detector; 

Fig. 8 is a block diagram illustrating the configuration of a 
feature point extractor; 

Fig. 9 is a schematic diagram used for explaining zero-cross 
points ; 

Fig. 10 is a schematic diagram used for explaining an image 
scanning direction; 

Figs. 11A and 11B are schematic diagrams used for explaining 
a feature point search range; 

Fig. 12 is a schematic diagram used for explaining an image 
composing method; 

Fig. 13 is a schematic diagram used for explaining a level 
conversion function; 
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Fig. 14 is a schematic diagram used for explaining histogram 
equalization; 

Fig. 15 is a block diagram illustrating the configuration of 
a video camera according to a second embodiment; 

Fig. 16 is a block diagram illustrating the configuration of 
a misregistration amount detector; 

Fig. 17 is a schematic diagram used for explaining the 
classification of zero-cross points; 

Fig. 18 is a block diagram illustrating the configuration of 
a video camera according to a third embodiment; 

Fig. 19 is block diagram illustrating the configuration of a 
misregistration amount detector; 

Figs. 20A and 20B are diagrams showing a feature point search 
range ; 

Fig. 21 is a block diagram illustrating the configuration of 
a video camera according to a fourth embodiment; 

Fig. 22 is a schematic diagram used for explaining an example 
of a color filter arrangement in a single-plate color camera; 

Fig. 23 is a schematic diagram used for explaining an output 
signal of the single-plate color camera; 

Figs. 24A and 24B are schematic diagrams used for explaining 
interpolation processing; 

Fig. 25 is a schematic diagram used for explaining a feature 
point scanning range; 

Fig. 26 is a block diagram illustrating the configuration of 
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a misregistration amount detector according to another embodiment; 

Fig. 27 is a schematic diagram used for explaining a 
confidence map; 

Fig. 28 is a block diagram illustrating the configuration of 
a misregistration amount detector according to another embodiment; 

Fig. 29 is a block diagram illustrating the configuration of 
a misregistration amount detector according to another embodiment; 
and 

Fig. 30 is a schematic diagram used for explaining an example 
of a color filter arrangement in a single-plate color camera. 



PEA I LED DESCRIPTION OF THE EMBODIMENT 

Preferred embodiments of this invention will be described 
with reference to the accompanying drawings: 
(1) First Embodiment 

In Fig. 4, a video camera, the configuration of which is 
generally designated by reference numeral 10, is adapted to 
capture two images having different amounts of exposure in a time 
division manner by use of a solid-state imager device (CCD: Charge 
Coupled Device) 12. As shown in Fig. 5, the CCD 12 accumulates 
and reads a charge in an arbitrary field period, in a manner 
similar to normal imaging, and subsequently accumulates and reads 
a charge again utilizing a vertical blanking period, to produce 
two images with different times of exposure within one field 
period. In this way, the CCD 12 produces two images with 
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different times of exposure, i.e., a long time exposed image xLl 
and a short time exposed image xSl, and sends the long time 
exposed image xLl to a memory 14, and sends the short time exposed 
image xSl to a memory 16. 

The memory 14 temporarily stores the long time exposed image 
xLl, and reads the long time exposed image xLl at a predetermined 
timing to send it to a limiter 18 and an image composer 20, 
respectively. The memory 16 temporarily stores the short time 
exposed image xSl, and reads the short time exposed image xSl at a 
predetermined timing to send it to a gain corrector 21 and a 
misregistration corrector 23, respectively. 

The gain corrector 21 corrects the short time exposed image 
xSl for a difference in the amount of exposure with the long time 
exposed image xLl at the time of imaging in accordance with the 
following equation: 

xS2(i, j) = xSl(i, j) x g (1) 

Then, the gain corrector 21 sends a resulting short time 
exposed image xS2 to a limiter 25. In the above equation, g is 
the ratio of exposure times between the long time exposed image 
xLl and the short time exposed image xSl at the time of imaging, 
and is given by the following equation: 
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where T long is an exposure time for the long time exposure , and T sho 
is an exposure time for the short time exposure. For reference , 
xSl(i, j) represents the value of a pixel at a position (i, j) on 
the short time exposed image xSl. 

The limiter 25 is provided for defining an upper limit for 
each pixel value on the short time exposed image xS2 which has 
been subjected to a gain correction. As given by the following 
equation: 



within respective pixels forming the short time exposed image xS2, 
a pixel value equal to or lower than a predetermined threshold 
value is outputted as it is, while a pixel value exceeding the 
predetermined threshold value is replaced with the threshold value 
T up which is outputted in place to produce a short time exposed 
image xS3 which is then sent to a misregistration amount detector 
27. 

The limiter 18 in turn is provided for defining an upper 
limit for each pixel value on the long time exposed image xLl. As 
is the case with the limiter 25, within respective pixels forming 
the long time exposed image xLl, a pixel value equal to or lower 
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than a predetermined threshold value is outputted as it is, while 
a pixel value exceeding the predetermined threshold value is 
replaced with the threshold value T up which is outputted in place 
to produce a long time exposed image xL2 which is then sent to a 
misregistration amount detector 27 . 

As shown in Fig. 6, the misregistration amount detector 27 
detects the amount of misregistration between the long time 
exposed image xL2 and the short time exposed image xS3 as a two- 
dimensional vector d= ss (dx / dy) which is sent to the misregistration 
corrector 23. Here, dx represents a horizontal component of the 
amount of misregistration; and dy a vertical component of the same. 

As illustrated in Fig. 7, the misregistration amount detector 
27 inputs the long time exposed image xL2 supplied from the 
limiter 18 to a feature point extractor 30A, and inputs the short 
time exposed image xS3 supplied from the limiter 25 to a feature 
point extractor 30B. The feature point extractor 30A extracts 
feature points from the long time exposed image xL2 to produce a 
feature point image pLl which is sent to a memory 32A. The 
feature point image pLl is a binary image in which each pixel has 
a value "0" or "1," with the pixel value "1" indicating that a 
corresponding pixel on the long time exposed image xL2 is a 
feature point, and the pixel value "0" indicating that a 
corresponding pixel is not a feature point. Employed herein as 
the feature point is, for example, an edge point at which a 
spatially abrupt change is present within the long time exposed 
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image xL2. 

Similarly , the feature point extractor 30B extracts feature 
points from the short time exposed image xS3 to produce a feature 
point image pSl which is sent to a memory 32B. Similar to the 
feature point image pLl, the feature point image pSl is a binary 
image in which each pixel has a value "0" or "1," with the pixel 
value "1" indicating that a corresponding pixel on the long time 
exposed image xS3 is a feature point, and the pixel value "0" 
indicating that a corresponding pixel is not a feature point. 

The feature point extractors 30A, 30B are similar in 
configuration, so that the feature point extractor 30A will only 
be described herein with respect to the circuit configuration with 
reference to Fig. 8. The feature point extractor 30A inputs the 
long time exposed image xL2 supplied from the limiter 18 (Fig. 4) 
to a second-order differentiation type filter 35. The second- 
order differentiation type filter 35 employs, for example, the 
following equation referred to as Laplacian 6: 



V 2 G(hk) - 

hmin ^ h ^ hmax 
kmin ^ k ^ kmax 

and performs a convolutional operation by the following equation: 
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je£3(i,;)= yx(i-h,j-k)xV 2 G(h,k) (5) 



h min Jlt£h max 



to find a second-order differentiated image xL3 which is sent to a 
binarizer 37. Here, O is a preset constant, and the larger the 0 
is, the more apt an edge of a lower frequency component to be 
detected. Also, h . , h , k . , k are constants for defining the 

nu.ii max min max 

span of the filter which implements Laplacian G. 

The binarizer 37 outputs the value "1" when a pixel value is 
"0 M or more and the value "0" when the pixel value is less than 
"0" at each of pixels on the second-order differentiated image xL3, 
to produce a binary image xL4 which is sent to a boundary point 
detector 39. The binary point detector 39 applies the binary 
image xL4 with neighborhood processing as given by the following 
equation: 



-b(ij)*b{i + dij + dj) 
-lsdi 3 dj^l 

others 



(6) 



to extract boundary points on a binary pattern to produce a 
feature point image pLl which is sent to the memory 32A (Fig. 7). 
Stated another way, the boundary point detector 39 outputs the 
value "1" only when pixels having different values exist within a 



neighborhood region, regarding that a feature point is present at 
a central position among them. The feature point extracted in 
this way is referred to as a zero-cross point which means a 
central point in an edge pattern at which the pixel value presents 
an abrupt change, as shown in Fig. 9. 

Turning back to Fig. 7, the memory 32A temporarily 
accumulates the feature point image pLl, and subsequently reads 
the feature point image pLl at a predetermined timing to send it 
to a confidence level calculator 45. Similarly, the memory 32B 
temporarily accumulates the feature point image pSl, and 
subsequently reads the feature point image pSl at a predetermined 
timing to send it to the confidence level calculator 45 . 

The confidence level calculator 45 calculates a confidence 
level which indicates the amount of correct misregistration for 
each of a plurality of previously provided misregistration amount 
candidates dn=(dxn, dyn) based on the feature point images pLl, 
pSl. The misregistration amount candidate dn may be given by 
inequalities as shown below, for example, by selecting a maximum 
value for the amount of misregistration which may be produced: 

-dxmax ^ dxn ^ dxmax 

-dymax ^ dyn ^ dymax (7) 

where dx indicates the absolute value of a maximum 

max 

misregistration amount in the horizontal direction; and dy 
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indicates the absolute value of a maximum misregistration amount 
in the vertical direction. 

Here, how to calculate the confidence level for each 
misregistration amount candidate dn will be explained in a 
specific manner. The confidence level calculator 45 first scans 
the feature point image pLl in an order as indicated in Fig. 10 to 
detect a position at which the pixel value indicates "1," i.e., 
the position of a feature point. Assuming that the position of a 
pixel corresponding to a feature point of interest in the feature 
point image pLl is at (i,j) (Fig. 11A) , the confidence level 
calculator 45 examines whether or not a feature point exists on 
each of positions (i+dxn, j+dyn) on the feature point image pSl, 
corresponding to a plurality of misregistration amount candidates 
dn=(dxn, dyn) . When the misregistration amount candidate dn is 
given by the inequalies such as those (7) mentioned above, a 
feature point search range on the feature point image pSl is 
represented by an area R as shown in Fig. 11B. 

As a result, when the confidence level calculator 45 detects 
that a feature point exists at a position (i+dxn, j+dyn) on the 
feature point image pSl, the confidence level calculator 45 
increments the confidence level for the misregistration amount 
candidate dn-(dxn, dyn) in accordance with the following equation: 

cmap(dn) = cmap (dn) + 1 (8) 
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where cmap(dn) is a confidence map for integrating the confidence 
level for each misregistration amount candidate, and is previously 
initialized as given by the following equation: 

cmap(dn) = 0 for all dn (9) 

In this way, the confidence level calculator 45 accumulates 
the confidence levels for all feature points on the feature point 
image pLl to produce the confidence map cmap (dn) (n=0, 1, 2,...) 
which is sent to a maximum value detector 47. 

The maximum value detector 47 detects a misregistration value 
candidate d having a maximum confidence level within the 

max 

confidence map cmap (dn), and sends this as a true misregistration 
amount & to the misregistration amount corrector 23 (Fig. 4) . For 
reference, if a plurality of misregistration amount candidates dn 
having the same confidence level exist, the maximum value detector 
47 selects the misregistration amount candidate dn that exhibits 
the smallest magnitude |dn| of the misregistration amount found by 
the following equation: 

|dn|2 = dxn2 + dyn2 (10) 

Also, when the maximum value d of a detected confidence 

max 

level is lower than a predetermined threshold value, i.e., does 
not satisfy a condition given by the following equation: 
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cmax > a x T C ount 

0.0 ^ a ^ 1.0 



(11) 



the maximum value detector 47 sets d=(0,0) so as not to perform a 
misregistration correction, regarding that a sufficiently 
confident misregistration amount cannot be available. In the 
above equation , & is a coefficient for determining a threshold 
value, and is a preset value, and T indicates the total number 

* * count 

of feature points extracted from the long time exposed image xL2. 

Turning back to Fig. 4, the misregistration corrector 23 
translates the short time exposed image xSl supplied from the 
memory 16 in accordance with the following equation: 

xS4(i,j) = xSl(i-dx,j-dy) (12) 

based on the misregistration amount vector d detected by the 
misregistration amount detector 27, to correct the misregistration 
with respect to the long time exposed image xLl, and sends a 
resultant short time exposed image xS4 to the image composer 20. 

The image composer 20 composes the long time exposed image 
xLl supplied from the memory 14 and the short time exposed image 
xS4 corrected by the misregistration corrector 23 to produce a 
single composite image xl having a wider dynamic range which is 
sent to a dynamic range compressor 49. 

A method of composing these long time exposed image xLl and 
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short time exposed image xS4 may involve accumulating a 
coefficient corresponding to the ratio of the amounts of exposure , 
with which respective images were captured, for each of the images f 
and subsequently outputting the respective images in turn 
according to thresholding to produce the composite image xl. In 
the following, the image composing method implemented by the image 
composer 20 will be explained in a specific manner with reference 
to Fig* 12. In Fig, 12, the horizontal axis represents the amount 
of incident light on the CCD 12, and the vertical axis represents 
a signal level of an output signal delivered from the CCD 12, i.e., 
a pixel level of a captured image. 

As shown in Fig. 12, the long time exposed image xLl is 
represented by a straight line having a large slope, but exhibits 
a constant signal level of the output signal due to saturation of 
the CCD 12 when the amount of incident light is equal to or larger 
than a predetermined level T^. The short time exposed image xS4 
in turn has a small slope, and the output signal delivered from 
the CCD 12 saturates with a larger amount of incident light than 
the long time exposed image xLl. 

Consequently, the image composer 20 first accumulates the 
coefficient g calculated by the aforementioned equation (2) for 
the output signal corresponding to the short time exposed image 
xS4 to match the slope of the straight line portion with that of 
the long time exposed image xLl. Subsequently, referring to the 
output signal corresponding to the long time exposed image xLl, 
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the image composer 20 selects the output signal of the long time 
exposed image xLl rather than the output signal of the short time 
exposed image xS4 when the output signal level of short time 
exposed image xS4 is equal to or lower than the predetermined 
threshold value T , and selects and outputs the output signal 
corresponding to the short time exposed image xS4 rather than the 
output signal of the long time exposed image xLl when the output 
signal level of short time exposed image xS4 is higher than the 
predetermined threshold value T w , thereby producing the composite 
image xl made from the long time exposed image xLl and the short 
time exposed image xS4. 

The image composing method implemented by the image composer 
20 is represented by the following equation: 



The coefficient g accumulated to the output signal corresponding 
to the short time exposed image xS4 is the ratio of the respective 
exposure times which is calculated by the aforementioned equation 
(2). In this way, if the ratio of exposing times is N times, the 
dynamic range of the composite image xl is extended by a factor of 
N. 

Next, the dynamic range compressor 49 compresses the dynamic 
range of the composite image xl in accordance with the data 
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processing capabilities of a transmission system, a display, a 
recorder and so on, not shown, disposed at stages subsequent 
thereto, and outputs a resulting compressed composite image yl to 
the outside. More specifically, the dynamic range compressor 49 
employs a method of converting a pixel level of each pixel in the 
composite image xl using a function having an input/output 
relationship as shown in Fig. 13 (hereinafter referred to as the 
"level conversion function"), as a method of compressing the 
composite image xl having a wide dynamic range (hereinafter 
referred to as the "level conversion"). 

In Fig. 13, the horizontal axis represents a pixel level 1 of 
the inputted composite image xl, and the vertical axis represents 
a pixel level T(l) of the compressed composite image yl which has 
undergone the level conversion processing. Linmax indicates a 
maximum pixel level that can be taken by each pixel in the 
inputted composite image xl, and Loutmax indicates a maximum pixel 
level which can be taken by each pixel in the compressed composite 
image yl outputted from the dynamic range compressor 49. In this 
level conversion, the full dynamic range is compressed, while 
ensuring the contrast in low and intermediate level ranges at the 
cost of the contrast in a high level range. 

Alternatively, the method of compressing the composite image 
xl having a wide dynamic range may involve adaptively changing the 
level conversion function in accordance with the frequency 
distribution of the pixel levels on the composite image xl. As a 
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representative example, a method called "histogram equalization" 
may be presented* Fig. 14 shows the principles of a method which 
implements the histogram equalization, where Fmax is a maximum 
value for a cumulative frequency which indicates the total number 
of pixels used to calculate the frequency. 

This compression method according to the histogram 
equalization first produces a frequency distribution H(l) with 
respect to the pixel level 1 of an inputted composite image xl, 
and next produces a cumulative frequency distribution C(l) in 
accordance with the following equation: 



Then, the compression method according to the histogram 
equalization normalizes the vertical axis of the cumulative 
frequency distribution to a level range possibly taken by a 
compressed composite image yl in accordance with the following 
equation to produce a level conversion function T(l): 
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r(/) = 



c(/) 



x Lout max 



(15) 



Fmax 



The compression method according to the histogram equalization 
utilizes the level conversion function T(l) to enable the 



compression of the full dynamic range while ensuring the contrast 
in a region composed by levels which present a high appearance 
frequency (a region having a large area). 

With the foregoing configuration, the misregistration amount 
detector 27 extracts feature points respectively from a long time 
exposed image xL2 and a short time exposed image xS3, captured 
with different exposure times , detects the misregistration amount 
candidate presenting the highest confidence level within a 
plurality of previously provided misregistration amount candidates 
dn based on the positional relationship of respective feature 
points extracted from the long time exposed image xL2 and 
respective feature points extracted from the short time exposed 
image xS3, and sends the detected misregistration amount candidate 
as a true misregistration amount d to the misregistration 
corrector 23. 

The misregistration corrector 23 corrects the misregistration 
of the short time exposed image xSl with respect to the long time 
exposed image xLl based on the misregistration amount d sent from 
the misregistration amount detector 27, and sends a resultant 
short time exposed image xS4 to the image composer 20. The image 
composer 20 composes the long time exposed image xLl and the 
misregistration corrected short time exposed image xS4 to produce 
a single composite image xl having a wide dynamic range. 

By thus composing the long time exposed image xLl and the 
short time exposed image xSl after correcting the misregistration 



therebetween, the image quality will never be degraded near the 
edges as observed in a composite image which would be composed 
with misregistration present between two images. 

According to the configuration described above, the 
misregistration of the short time exposed image xSl is corrected 
with respect to the long time exposed image xLl, and then the long 
time exposed image xLl and the misregistration corrected short 
time exposed image xS4 are composed to produce the composite image 
xl to avoid a degraded image quality which would otherwise be 
found when two images are composed without correction, thereby 
making it possible to produce a natural composite image suitable 
for an object to be imaged. 

(2) Second Embodiment 

In Fig* 15 , in which parts corresponding to those in Fig. 4 
are designated by the same reference numerals , a video camera 
according to a second embodiment, generally designated by 
reference numeral 60, is configured in a similar manner to the 
video camera 10 according to the first embodiment except for the 
configuration of a misregistration amount detector 62. 

Fig. 16, in which parts corresponding to those in Fig. 7 are 
designated by the same reference numerals, illustrates the 
configuration of the misregistration amount detector 62. This 
misregistration amount detector 62 is adapted to extract four 
types of feature points. More specifically, zero-cross points as 
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previously shown in Fig. 9 are classified into feature points PI 
to P4 as shown in Fig. 17, based on the signs of pixel values 
possessed by pixels surrounding the respective zero-cross points. 

The following description will be directed to a 
misregistration amount detection performed by the misregistration 
amount detector 62. The misregistration amount detector 62 inputs 
a long time exposed image xL2 supplied from the limiter 18 to 
feature point extractors 64A x to 64A 4 , and inputs a short time 
exposed image xS3 supplied from the limiter 25 to feature point 
extractors 64B to 64B . 

1 4 

The feature point extractor 64A A extracts only feature points 
PI from the long time exposed image xL2 to produce a feature point 
image pLll which is sent to a memory 66A x . The feature point 
extractor 64B x in turn extracts only the feature point Pi from the 
short time exposed image xS3 to produce a feature point image sPll 
which is sent to a memory 66B 1 . Similarly, the feature point 
extractors 64A 2 to 64A 4 extract only the feature points P2 to P4 
corresponding thereto from the long time exposed image xL2 to 
produce feature point images pLl2 to pl*14 which are sent to 
corresponding memories 66A 2 to 66A 4 , respectively. The feature 
points extractors 64B 2 to 64B 4 in turn extract only feature points 
P2 to P4 corresponding thereto from the short time exposed image 
xS3 to produce feature point images pS12 to pS14 which are sent to 
corresponding memories 66B 2 to 66B 4 , respectively. 

The memory 66A X temporarily stores the feature point image 
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pLll, and subsequently reads -the feature point image pLll at a 
predetermined timing to send it to a confidence level calculator 
68^ The memory 66B if in turn, temporarily stores the feature 
point image pSll, and subsequently reads the feature point image 
pSll at a predetermined timing to send it to the confidence level 
calculator 68^ Similarly, the memories 66A 2 to 66A 4 temporarily 
store the feature point images pL12 to pLl4, respectively, and 
read them at a predetermined timing to send them to corresponding 
confidence level calculators 68 2 to 68 4 , respectively. The 
memories 66B 2 to 66B 4 temporarily store the feature point images 
pS12 to pS14, respectively, and read them at a predetermined 
timing to send them to corresponding confidence level calculators 
68 2 to 68^, respectively. 

The confidence level calculator 68 i produces a confidence map 
cmapl(dn) based on the feature point images pLll, pSll and sends 
the confidence map cmapl(dn) to an adder 70. Similarly, the 
confidence level calculators 68 2 to 68 4 produce confidence map 
cmap2(dn) to cmap4(dn), respectively, and send them to the adder 
70. The adder 70 adds the confidence maps cmapl(dn) to cmap4(dn) 
for each element in accordance with the following equation: 



_4 

cmai 



ip{dri) » ^ cmapN(dri) (16) 



to produce a single confidence map cmap(dn) which is sent to a 
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maximum value detector 47 . The maximum value detector 47 detects 
a misregistration amount candidate d having a maximum confidence 
level within the confidence map cmap(dn) , and sends this as a true 
misregistration amount d to a misregistration amount corrector 
(Fig. 15). 

In the foregoing configuration,, the misregistration amount 
detector 62 extracts four types of feature points PI to P4, 
respectively from the long time exposed image xL2 and the short 
time exposed image xS3, captured with different exposure times, 
and calculates the confidence level for a plurality of 
misregistration amount candidates dn for each type of the 
extracted feature points. Then, the misregistration amount 
detector 62 calculates the sum of the confidence levels calculated 
for each type of the feature points for each misregistration 
amount candidate dn, and detects the misregistration amount 
candidate dn having the highest confidence level within the thus 
calculated confidence levels for the plurality of misregistration 
amount candidates dn, and sends this as a true misregistration 
amount d to the misregistration corrector 23. 

The misregistration corrector 23 corrects the misregistration 
of the short time exposed image xSl with respect to the long time 
exposed image xLl based on the misregistration amount d sent from 
the misregistration amount detector 27, and sends a resultant 
short time exposed image xS4 to the image composer 20. The image 
composer 20 composes the long time exposed image xLl and the 
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misregistration corrected short time exposed image xS4 to produce 
a single composite image xl having a wide dynamic range. 

By thus composing the long time exposed image xLl and the 
short time exposed image xSl after correcting the misregistration 
therebetween, the image quality will never be degraded near the 
edges as observed in a composite image which would be composed 
with misregistration present between two images. 

According to the configuration described above, the 
misregistration of the short time exposed image xSl is corrected 
with respect to the long time exposed image xLl, and then the long 
time exposed image xLl and the misregistration corrected short 
time exposed image xS4 are composed to produce the composite image 
xl to avoid a degraded image quality which would otherwise be 
found when two images are composed without correction, thereby 
making it possible to produce a natural composite image 
corresponding to an object to be imaged. 

In addition , since plural types of feature points are 
extracted from the long time exposed image xL2 and the short time 
exposed image xS3 to detect the misregistration amount £ using the 
thus extracted feature points of the plural types , the 
misregistration amount can be more accurately detected as compared 
with the misregistration amount detector 27 in the first 
embodiment . 

(3) Third Embodiment 
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In Fig. 18 , in which parts corresponding to those in Fig. 4 
are designated by the same reference numerals, a video camera, the 
configuration of which is generally designated by reference 
numeral 100, is configured in a manner similar to the video camera 
10 according to the first embodiment except for the configuration 
of horizontal integrators 102, 103, vertical integrators 104, 105, 
misregistration amount detectors 108, 109, a misregistration 
corrector 111 . 

The video camera 100 supplies the horizontal integrator 102 
and the vertical integrator 104 with a long time exposed image xL2 
sent from the limiter 18, and supplies the horizontal integrator 
103 and the vertical integrator 105 with a short time exposed 
image xS3 sent from the limiter 25. The horizontal integrator 102 
integrates pixel values at respective vertical positions i of the 
long time exposed image xL2 in the horizontal direction in 
accordance with the following equation: 

Nx-l 

xLHl(j)=^xL2{i,i) (17) 

j=0, 1, 2, Ny-1 

to produce a one-dimensional data array xLHl which is sent to the 
misregistration amount detector 108. In equation (17), N x and N x 
are the numbers of pixels in the horizontal direction and in the 



-27- 



vertical direction of the image. 

The vertical integrator 104 integrates pixel values at 
respective horizontal positions i of the long time exposed image 
xL2 in the vertical direction in accordance with the following 
equation: 



to produce a one-dimensional data array xLVl which is sent to the 
misregistration amount detector 109. Similarly , the horizontal 
integrator 103 produces a one-dimensional data array xSHl from the 
short time exposed image xS3, and sends the one-dimensional data 
array xSHl to the misregistration amount detector 108. Together 
with this, the vertical integrator 105 produces a one-dimensional 
data array xSVl from the short time exposed image xS3, and sends 
the one-dimensional data array xSVl to the misregistration amount 
detector 109. 

The misregistration amount detector 108 detects a vertical 
component dy of a misregistration amount based on the one- 
dimensional data array xLHl produced from the long time exposed 
image xL2 and the one-dimensional data array xSHl produced from 
the short time exposed image xS3, and sends the detected vertical 




(18) 



i=0, 1, 2, 



• • . z 



Nx-1 
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component dy to the misregistration corrector 111. Together with 
this, the misregistration amount detector 109 detects a horizontal 
component dx of the misregistration amount based on the one- 
dimensional data array xLVl produced from the long time exposed 
image xL2 and the one-dimensional data array xSVl produced from 
the short time exposed image xS3, and sends the horizontal 
component dx to the misregistration corrector 111. 

It should be noted that the misregistration amount detectors 
108, 109 are similar in configuration, so that the following 
description will be directed only to the configuration of the 
misregistration amount detector 109 with reference to Fig. 19. 
The misregistration amount detector 109 inputs the one-dimensional 
data array xLVl produced from the long time exposed image xL2 to a 
feature point extractor 115A, and inputs the one-dimensional data 
array xSVl produced from the short time exposed image xS3 to a 
feature point extractor 115B. 

The feature point extractor 115 A extracts feature points in 
the one-dimensional data array xLVl to produce a one-dimensional 
feature point array pLVl which is sent to a memory 117A. The 
feature point called herein is the zero-cross point shown in Fig. 
9, and the one-dimensional data array xLVl is an array of binary 
data each indicating whether or not a feature point exists at a 
position i. Similarly, the feature point extractor 115B extracts 
feature points in the one-dimensional data array xSVl to produce a 
one-dimensional data array pSVl which is sent to a memory 117B. 
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The memory 117A temporarily stores the feature point array 
pLVl, and subsequently reads the feature point array pLVl at a 
predetermined timing to send it to a confidence level calculator 
119. Similarly, the memory 117B temporarily stores the feature 
point array pSVl, and subsequently reads the feature point array 
pSVl at a predetermined timing to send it to the confidence level 
calculator 119. 

The confidence level calculator 119 calculates a confidence 
level as a correct misregistration amount for a horizontal 
component dx of each of a plurality of previously provided 
misregistration amount candidates, based on the feature point 
arrays pLVl, pSVl. The horizontal component dx of this 
misregistration amount candidate is given, for example, by the 
following inequality: 

-dxmax S dxn S dxmax (19) 

by selecting a maximum value for the horizontal component of a 
misregistration amount which could occur. 

Here, how to calculate the confidence level for the 
horizontal component dx of each misregistration amount candidate 
will be explained in a specific manner. The confidence level 
calculator 119 first scans the feature point array pLVl to detect 
the position of a point at which the value indicates "1," i.e., 
the position of a feature point. Assuming that the position of a 
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feature point of interest on the feature point array pLVl is at i 
(Fig. 20A), the confidence level calculator 119 examines whether 
or not a feature point exists at a position i+dx on the feature 
point array pSVl, corresponding to a horizontal component dx of 
each of a plurality of misregistration amount candidates. When 
the horizontal component dx of the misregistration amount 
candidate is given by the inequality such as that (19) mentioned 
above, a feature point search range on the feature point array 
pSVl is represented by an area R as shown in Fig. 20B. 

As a result, when the confidence level calculator 119 detects 
that a feature point exists at a position i+dx on the feature 
point array pSVl, the confidence level calculator 119 increments 
the confidence level for the horizontal component dx of the 
misregistration amount candidate. In this way, the confidence 
level calculator 119 integrates the confidence levels for all 
feature points on the feature point array pLVl to produce a 

confidence map cmap(dxn) (n=0, 1, 2, ) which is sent to a 

maximum value detector 121. The maximum value detector 121 
detects a horizontal component dx^ of a misregistration amount 
candidate which has a maximum confidence level within the 
confidence map cmap(dxn) , and sends this as a horizontal component 
dx of a true misregistration amount to the misregistration 
corrector 111 (Fig. 18). 

Turning back to Fig. 18, the misregistration corrector 111 
translates the short time exposed image xSl based on the 
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horizontal component dx of the misregistration amount supplied 
from the misregistration amount detector 109 and the vertical 
component dy of the misregistration amount supplied from the 
misregistration amount detector 108 to correct the misregistration 
of the short time exposed image xSl with respect to the long time 
exposed image xLl, and sends a resultant short time exposed image 
xS4 to the image composer 20. 

In the foregoing conf iguration, the horizontal integrators 
102, 103 integrate pixel values for each pixel array in the 
horizontal direction of a long time exposed image xL2 and a short 
time exposed image xS3, respectively, to produce one-dimensional 
vertical data arrays xLHl, xSHl which are sent to the 
misregistration amount detector 108. Together with this, the 
vertical integrators 104 , 105 integrate pixel values for each 
pixel arrays in the vertical direction of the long time exposed 
image xL2 and the short time exposed image xS3, respectively, to 
produce one-dimensional horizontal data arrays xLVl, xSVl which 
are sent to the misregistration amount detector 109. 

The misregistration amount detector 108 extracts feature 
points from the one-dimensional vertical data arrays xLHl, xSHl, 
detects a vertical component of the misregistration amount 
candidate presenting the highest confidence level within vertical 
components dyn of a plurality of previously provided 
misregistration amount candidates based on the positional 
relationship of the respective feature points in the one- 
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dimensional data array xLHl and the respective feature points in 
the one-dimensional data array xSHl extracted as mentioned, and 
sends this as a true misregistration amount dy to the 
misregistration corrector 111. Similarly, the misregistration 
amount detector 109 extracts feature points from the one- 
dimensional horizontal data arrays xLVl, xSVl, detects a 
misregistration amount dx based on the positional relationship of 
the respective feature points in the one-dimensional data array 
xLVl and the respective feature points in the one-dimensional data 
array xSHl extracted as mentioned, and sends the detected 
misregistration amount dx to the misregistration corrector 111. 

The misregistration amount corrector 111 corrects the 
misregistration of the short time exposed image xSl with respect 
to the long time exposed image xLl based on the misregistration 
amounts dx, dy sent from the misregistration amount detectors 109, 
108, and sends a resultant short time exposed image xS4 to the 
image composer 20. The image composer 20 composes the long time 
exposed image xLl and the misregistration corrected short time 
exposed image xS4 to produce a signal composite image xl having a 
wide dynamic range. 

By thus composing the long time exposed image xLl and the 
short time exposed image xSl after correcting the misregistration 
therebetween, the image quality will never be degraded near the 
edges as observed in a composite image which would be composed 
with misregistration present between two images. 
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According to the configuration described above , the 
misregistration of the short time exposed image xSl is corrected 
with respect to the long time exposed image xLl, and then the long 
time exposed image xLl and the misregistration corrected short 
time exposed image xS4 are composed to produce the composite image 
xl to avoid a degraded image quality which would otherwise be 
found when two images are composed without correction , thereby 
making it possible to produce a natural composite image 
corresponding to an object to be imaged. 

Also, by detecting the misregistration amount of the short 
time exposed image xSl with respect to the long time exposed image 
xLl for each of the horizontal component and the vertical 
component , it is possible to further reduce the amount of 
operations and the circuit scale associated with the detection of 
misregistration . 

(4) Fourth Embodiment 

In Fig. 21, in which parts corresponding to those in Fig. 4 
are designated by the same reference numerals, a video camera 
according to a fourth embodiment , generally designated by 
reference numeral 120, comprises a color array filter as shown in 
Fig. 22 (complementary color diced filter) positioned in front of 
a CCD 122 to capture a color image through this filter to produce 
a long time exposed image xLlO and a short time exposed image xSlO 
represented by output signals, as shown in Fig. 23, in which a 



-34- 



frequency modulated color signal is multiplexed on a luminance 
signal . 

In addition, the video camera 120 includes luminance 
separators 124 , 125 previous to a gain corrector 21 and a limiter 
18 , respectively , to extract luminance information in images. 
Specifically, the luminance separator 124 extracts luminance 
information xLyl from the long time exposed image xLlO in 
accordance with the following equation: 



xLyl = LPFy (XL10) (20) 

From the luminance information xLyl, the limiter 18 extracts 
luminance information xLy2 which is sent to a misregistration 
amount detector 27, In equation (20) , LPF y , () indicates a low pass 
filter for separating the luminance. Similarly , the luminance 
separator 125 extracts luminance information xSyl from the short 
time exposed image xSlO, and luminance information xSy2 is 
produced sequentially through the gain corrector 21 and a limiter 
25 , and sent to the misregistration amount detector 27 . 

The misregistration amount detector 27 detects a 
misregistration amount d between the luminance information Ly2 
produced from the long time exposed image xLlO and the luminance 
information xSy2 produced from the short time exposed image xSlO, 
and sends the misregistration amount d to a misregistration 
corrector 127 . The misregistration corrector 127 corrects the 
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misregistration of the short time exposed image xSlO based on the 
misregistration amount 

In this event, since the value of each pixel forming part of 
the long time exposed image xLlO and the short time exposed image 
xSlO is modulated by the color filter (Fig. 22), a pixel xLlO 
(i,j) on the long time exposed image xLlO and a pixel xSlO (i-dx, 
j-dy) on the short time exposed image xSlO, corresponded by the 
detected misregistration amount d, do not always correspond to the 
same color filter. In other words, the misregistration may result 
in failed correspondence in color. To solve this problem, the 
misregistration amount corrector 127 corrects the misregistration 
in accordance with the following equation: 

xSll(i,j) = xSip(i-dx,j-dy) (21) 

and sends a resultant short time exposed image xSll to an image 
composer 20. In equation (21), xSip( i-dx, j-dy) indicates the 
value of a pixel at a position (i-dx, j-dy) on the short time 
exposed image xSlO, corresponding to the same color filter as the 
long time exposed image xLlO (i,j), calculated from the values of 
neighboring pixels. 

For example, as shown in Figs. 24A and 24B, assuming that a 
pixel on the long time exposed image xLlO (i,j) corresponds to a 
Ye filter with a misregistration amount d=(l,l), xSip(i-dx, j-dy) 
is interpolated in accordance with the following equation: 
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xsip(i',j') = 



xS10(V-X J M -1) + xS10(f '+1, J -1) + xS10(V-l j'-l) + xS10(V+% /'+!) 

4 

i'=i-dx, j'=j-dy 

(22) 

In other words, the misregistration corrector 127 corresponds to 
the same color filter as the long time exposed image xLlO (i,j), 
and performs linear interpolation using the value of pixel nearest 
from a position (i-dx, j-dy). 

In the foregoing configuration, the misregistration amount 
detector 27 extracts feature points from the luminance information 
xLy2 produced from the long time exposed image xLlO and the 
luminance information xSy2 produced from the short time exposed 
image xSlO, captured with different exposure times, detects the 
misregistration amount candidate presenting the highest confidence 
level within a plurality of previously provided misregistration 
amount candidates dn based on the positional relationship of the 
respective feature points extracted from the luminance information 
xLy2 and the respective feature points extracted from the 
luminance information xSy2, and sends the detected misregistration 
amount candidate as a true misregistration amount £ to the 
misregistration corrector 127 . 
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The misregistration corrector 127 corrects the 
misregistration of the short time exposed image xSlO with respect 
to the long time exposed image xLlO based on the misregistration 
amount sent from the misregistration amount detector 27, and sends 
a resultant short time exposed image xSll to the image composer 20 
The image composer 20 composes the long time exposed image xLlO 
and the misregistration corrected short time exposed image xSll to 
produce a single composite image xl having a wide dynamic range. 

By thus composing the long time exposed image xLlO and the 
short time exposed image xSlO after correcting the misregistration 
therebetween , the image quality will never be degraded near the 
edges as observed in a composite image which would be composed 
with misregistration present between two images. 

According to the configuration described above, the 
misregistration of the short time exposed image xSlO is corrected 
with respect to the long time exposed image xLlO, and then the 
long time exposed image xLlO and the misregistration corrected 
short time exposed image xSll are composed to produce the 
composite image xl to avoid a degraded image quality which would 
otherwise be found when two images are composed without correction 
thereby making it possible to produce a natural composite image 
corresponding to an object to be imaged. 

In addition , since the luminance information xLy2 and the 
luminance information xSy2 are extracted from the long time 
exposed image xLlO and the short time exposed image xSlO, 
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respectively, to detect a misregistration amount d from the 
luminance information xLy2 and the luminance information xSy2, it 
is possible to precisely correct misregistration even for a color 
image . 

(5) Other Embodiments 

While the foregoing first to fourth embodiments have been 
described in connection with an implementation in which the gain 
is corrected within the image composer 20 , the present invention, 
however, is not limited to such gain correction. Alternatively, a 
gain corrector can be disposed previous to the memory 16 for 
storing the short time exposed image xSl to omit the gain 
correction performed with the image composer 20. 

Also, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
short time exposed images xSl, xSlO are corrected for 
misregistration, the present invention is not limited to the 
correction to the short time exposed images. Alternatively, the 
long time exposed images xLl, xLlO can be corrected for 
misregistration . 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
gain correction is performed on the short time exposed images xS4, 
xSll, the present invention is not limited to such gain correction. 
Alternatively, the gain correction can be performed on the long 
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time exposed images xLl, xLlO. In this case, the value of each 
pixel on the long time exposed images xLl, xLlO can be divided by 
an exposure time ratio g calculated by the aforementioned equation 
(2). 

Further , while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
amount of misregistration is detected after the gain correction is 
performed on the short time exposed images xSl, xSlO, the present 
invention is not limited to such detection accompanied with the 
gain correction. Alternatively f the amount of misregistration can 
be detected without gain correction if pixel positions independent 
of the amount of exposure are used as feature points, as the zero- 
cross points shown in Fig. 9. 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
confidential calculator 45, 68 or 119 scans the entire feature 
point image pLl, pLll to pL14 corresponding to the long time 
exposed image xL2 to detect feature points and integrate the 
confidence levels, the present invention is not limited to the 
scanning of the entire feature point image. Alternatively, since 
feature points positioned near edges of an image are not likely to 
appear in the feature point images pSl, pSll to pS14 corresponding 
to the short time exposed image xS3 due to the influence of 
misregistration, a range in which feature points are scanned can 
be limited on the feature point images pLl, pLll to pL14, as shown 



in Fig. 25. 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which a 
search is conducted to see whether or not a corresponding point 
exists on the feature point image pSl, pSll to pS14 produced from 
a short time exposed image xS3 for each feature point on the 
feature point image pLl, pLll to pL14 produced from a long time 
exposed image xL2, the present invention is not limited to such a 
search. Alternatively, a corresponding point on the feature point 
image pLl, pLll to pLl4 can be searched for each feature point on 
the feature point image pSl, pSll to pS14. 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
confidence level calculator 45 calculates a confidence level by 
scanning the feature point image pLl, pLll to pL14 to detect 
feature points thereon, and examining whether or not corresponding 
feature points on the feature point image pSl, pSll to pS14 exist 
at positions at which the detected feature positions are corrected, 
using a misregistration amount candidate dn, the present invention 
is not limited to the thus calculated confidence level. 
Alternatively, as expressed by the following equation: 



cmap(dn) 



Nx-l Ny-1 




(23) 
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the entire feature point image pSl, pSll to pS!4 can be translated 
by a currently intended misregistration amount candidate dn, and 
the total number of feature points that match the feature points 
on the corresponding feature point image pLl, pLll to pLl4 in 
position can be employed as a confidence level cmap (dn) for the 
misregistration amount candidate dn. In equation (23), N x and N y 
are the numbers of pixels in the horizontal direction and the 
vertical direction, respectively, on the feature point image pSl, 
pSll to pS14. For reference, for points (i-dnx, j-dny) which do 
not exist on the feature point image pSl, pSll to pS14, the 
addition indicated by the aforementioned equation (23) is not 
performed. 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
maximum value detector 47 detects a misregistration amount 
candidate d having a maximum confidence level within the 

max *■* 

confidence map cmap (dn) sent from the confidence level calculator 
45, and employs d ma x as a true misregistration amount &, the 
present invention is not limited to this method of detecting the 
misregistration amount. Alternatively, as illustrated in a 
misregistration amount detector 200 in Fig. 26, a smoother 202 can 
be provided between the confidence level calculator 45 and the 
maximum value detector 47, such that, taking advantage of the fact 
that the confidence map cmap (dn) can be regarded as two- 
dimensional image data cmap (dx, dy) as shown in Fig. 27 which has 



the values of confidence levels as pixel values, the smoother 202 
applies mean-value filtering as expressed by the following 
equation: 

cmap' (dx,dy) 



M £ 
2 2 

= ^cmap{dx + di,dy + dj) (24) 

dt= dj~ 

2 2 



to the image data cmap (dx, dy) to produce a confidence map 
cmap'(dn), in which case a precise amount of misregistration can 
be detected with much higher stability. 

Further, while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
confidence level calculator 45 searches a corresponding point on 
the feature point image pSl for each feature point on the feature 
point image pLl to produce the confidence map cmap (dn), the 
present invention is not limited to such a manner of producing the 
confidence map cmap (dn) - Alternatively , as a confidence level 
calculator 206 in a misregistration amount detector 204 
illustrated in Fig. 28, a higher confidence level can be added as 
corresponding pixel values are closer on the inputted feature 
point image pLl and feature point image pS, in which case the 
amount of misregistration can be detected with much higher 
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stability, in this event, the confidence level calculator 206 can 
find the confidence map cmap(dn) in accordance with the following 
equation: 



cmap(dn) = cmap(dn) + s(dn) (25) 

where s(dn) is the amount representing the closeness of pixel 
values on the feature point image pLl and the feature point image 
pS, and is calculated, for example, in accordance with the 
following equation: 

s(dn) 

\xL f, (iJ)'xS n (i-dnxJ-dny)\ < TO 

* IrO * \xL"(i,j) - xS"(i-dnx,j-dny)\ < Tl 
0... 

Tl ^ \xL"(ij) - xS"(i-dnxj-dny)\ 

(26) 

Further , while the foregoing first to fourth embodiments have 
been described in connection with an implementation in which the 
long time exposed image xLl and the short time exposed image xS4 
are composed to produce a signal composite image, the present 
invention is not limited to the composition of two images, but 
three or more images with different amounts of exposure can be 
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composed. In this case, any of a plurality of images is assigned 
as a reference image, such that a misregistration amount detector 
and a misregistration corrector are provided to correspond to 
images other than the reference image and correct misregistration 
of the images with respect to the reference image. 

Further, the foregoing first to fourth embodiments have been 
described in connection with an implementation in which the long 
time exposed image xLl and the short time exposed image xS4 are 
composed to produce a signal composite image. The present 
invention, however, is not limited to the composition of two 
images, and for composing images with three or more different 
exposure times, the composition processing in accordance with the 
aforementioned equation (13) can be repeated in order from the 
image with the longest exposure time. 

In the foregoing second embodiment, the misregistration 
amount detector 62 extracts four types of feature points Pi to P4 
from the long time exposed image xL2 and the short time exposed 
image xS3, respectively. The present invention, however, is not 
limited to the four types of feature points, but essentially, 
plural types of feature points can be extracted from the images. 

Also, in the foregoing second embodiment, the feature point 
images pLll to pLl4 and pSll to pS14 are written into the 
corresponding memories 66A 1 to 66A 4 and 66B 1 to 66B 4 , respectively. 
The present invention, however, is not limited to such a manner of 
writing the feature point images. Alternatively, as is the case 
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of a misregistration amount detector 210 illustrated in Fig. 29, 
feature point images pLll to pLl4 sent from feature point 
extractors 74A 2 to 74A 4 can be written into a memory 212 A, while 
feature point images pSll to pS14 sent from feature point 
extractors 74B i to 74B 4 can be written into a memory 212B to 
produce multi-value feature point extraction images pL20, pS20, in 
which a pixel can take a value from zero to three , and the feature 
point extraction images pL20, pS20 are supplied to the confidence 
level calculator 214 to detect the amount of misregistration. 

Further , while in the foregoing fourth embodiment, the 
luminance separators 124, 125 are provided previous to the limiter 
18 and the gain corrector 21 , respectively, the present invention 
is not limited to this configuration. Alternatively, when a zero- 
cross point extracted by the feature point extractor 30A shown in 
Fig. 8 is used as the feature point, a filter which additionally 
has the characteristic of removing or reducing the modulation 
introduced by a color filter can be used in place of the second- 
order differentiation type filter 35 to omit the luminance 
separators 124 , 125 . 

Also, while in the foregoing fourth embodiment, the 
complementary color diced filter is placed in front of the CCD 122, 
the present invention is not limited to the placement of the diced 
filter, but a variety of other color filters can be placed in 
front of the CCD 112, such as a primary color stripe filter, in 
which case, appropriate interpolation processing can be performed 
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in accordance with -the arrangement of placed color filters. 

Further , while in the foregoing fourth embodiment, the 
misregistration corrector 23 performs the interpolation processing 
to eliminate failed correspondence in color, the present invention 
is not limited to such an approach of elimination. Alternatively, 
misregistration amount candidates can be limited to eliminate 
failed correspondence in color due to the misregistration 
correction. More specifically, assuming that repeating periods in 
a color filter array in the horizontal direction and the vertical 
direction are Th, Tv, respectively, the value possibly taken by a 
misregistration amount candidate dn=(dnx,dny) is limited by the 
following equation: 

dnx = Th X m 

dny « Tv X m (27) 

where m is an arbitrary integer value. A misregistration amount d 
detected by this limitation is an integer multiple of the 
repeating period of the color filter, so that the failed 
correspondence in color can be eliminated. For example, when a 
color filter as shown in Fig. 30 is employed, it is possible to 
detect the misregistration amount in units of two pixels both in 
the horizontal direction and the vertical direction. 

Further, while the foregoing first and fourth embodiments 
have been described for a video camera to which the present 
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invention is applied , the present invention is not limited to the 
video camera but can be widely applied to a variety of other 
apparatus equipped with an imager device such as a still camera, a 
surveillance camera, a car-equipped camera, and so on. 

According to the present invention as described above, a 
misregistration amount of each of images except for a reference 
image is detected with respect to the reference image, 
misregistration of each of the images except for the reference 
image is corrected with respect to the reference image based on 
the misregistration amount, and then the reference image is 
composed with each of misregistration corrected images other than 
the reference image, thereby making it possible to avoid a 
degraded image quality which would otherwise be produced when a 
plurality of images are composed without correction, and 
consequently produce a natural composite image corresponding to an 
object to be imaged. 

While there has been described in connection with the 
preferred embodiments of the invention, it will be obvious to 
those skilled in the art that various changes and modifications 
may be aimed, therefore, to cover in the appended claims all such 
changes and modifications as fall within the true sprit and scope 
of the invention. 
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WHAT I S CL AIMED I S; 

1. An image capturing apparatus for capturing a plurality of 
images with different amounts of exposure to compose said images 
into a single composite image f said image capturing apparatus 
comprising: 

misregistration amount detecting means for detecting the 
amounts of misregistration of respective images except for a 
reference image, said reference image selected from said plurality 
of images, with respect to said reference image; 

misregistration correcting means for correcting the 
misregistration of the respective images except for said reference 
image with respect to said reference image based on said 
misregistration amounts; and 

image composing means for composing said reference image and 
all of the respective misregistration corrected images except for 
said reference images. 

2. The image capturing apparatus according to claim 1/ wherein 
said misregistration amount detecting means comprises: 

feature point extracting means for extracting feature 
points from said reference image and the respective images 
except for said reference image, respectively; 

confidence level calculating means for calculating a 
confidence level for each of the images except for said 
reference image as said misregistration amount for each of a 
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plurality of previously provided misregistration amount 
candidates , based on a positional relationship of the feature 
points in said reference image and the feature points in the 
respective images except for said reference image; and 

maximum value detecting means for detecting said 
misregistration amount candidate which presents a maximum 
confidence level within said misregistration amount 
candidates as said misregistration amount in each of the 
images except for said reference image. 

3. The image capturing apparatus according to claim 2, wherein 
said confidence level calculating means corrects 

misregistration between a feature point in said reference image 
and a feature point in an image except for said reference image 
using said misregistration amount candidate, and counts the number 
of positions of said feature points which correspond to each other 
to calculate said confidence level, thereby calculating said 
confidence level in each of said plurality of misregistration 
amount candidates. 

4. The image capturing apparatus according to claim 2, wherein 
said feature point extracting means extracts plural types of 

feature points from said reference image and each of the images 
except for said reference image; and 

said confidence level calculating means calculates the 



-50- 



confidence level for each of the images except for said reference 
image as said misregistration amount for each of said plurality of 
previously provided misregistration amount candidates based on a 
positional relationship of said plural types of feature points 
extracted from said reference image and said plural types of 
feature points extracted from each of the images except for said 
reference image. 

5. The image device according to claim 2, wherein 

said feature point is a pixel which forms an edge at which a 
pixel value presents an abrupt change. 

6. The image capturing apparatus according to claim 2 
comprising: 

integrating means for integrating pixel values for each pixel 
train in the horizontal direction to produce a data array in the 
vertical direction, and for integrating pixel values for each 
pixel train in the vertical direction to produce a data array in 
the horizontal direction , for each of said plurality of images, 

wherein said feature point extracting means extracts feature 
points in said reference image and feature points in each of the 
images except for said reference image based on the respective 
data arrays in the vertical and horizontal directions. 

7 . The image capturing apparatus according to claim 2 , wherein 
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said feature point extracting means extracts feature points 
in said reference image and feature points in each of the images 
except for said reference image based on luminance information 
included in said plurality of images. 

8. An imaging method for capturing a plurality of images with 
different amounts of exposure to compose said images into a single 
composite image, said imaging method comprising the steps of: 

detecting the amounts of misregistration of respective images 
except for a reference image , said reference image selected from 
said plurality of images, with respect to said reference image; 

correcting the misregistration of the respective images 
except for said reference image with respect to said reference 
image based on said misregistration amounts; and 

composing said reference image and all of the respective 
misregistration corrected images except for said reference images. 

9. The imaging method according to claim 8, comprising the steps 
of: 

extracting feature points from said reference image and the 
respective images except for said reference image, respectively; 

calculating a confidence level for each of the images except 
for said reference image as said misregistration amount for each 
of a plurality of previously provided misregistration amount 
candidates, based on a positional relationship of the feature 
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points in said reference image and the feature points in the 
respective images except for said reference image; and 

detecting said misregistration amount candidate which 
presents a maximum confidence level within said misregistration 
amount candidates as said misregistration amount in each of the 
images except for said reference image. 

10. The imaging method according to claim 9, wherein 
misregistration between a feature point in said reference 

image and a feature point in an image except for said reference 
image is corrected by using said misregistration amount candidate, 
and the number of positions of said feature points which 
correspond to each other is counted to calculate said confidence 
level, thereby said confidence level in each of said plurality of 
misregistration amount candidates is calculated. 

11. The imaging method according to claim 9, comprising the steps 
of: 

extracting plural types of feature points from said reference 
image and each of the images except for said reference image; and 
calculating the confidence level for each of the images 
except for said reference image as said misregistration amount for 
each of said plurality of previously provided misregistration 
amount candidates based on a positional relationship of said 
plural types of feature points extracted from said reference image 
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and said plural types of feature points extracted from each of the 
images except for said reference image. 

12. The imaging method according to claim 9, wherein 

said feature point is a pixel which forms an edge at which a 
pixel value presents an abrupt change. 

13. The imaging method according to claim 9, comprising the steps 
of: 

integrating pixel values for each pixel train in the 
horizontal direction to produce a data array in the vertical 
direction, and integrating pixel values for each pixel train in 
the vertical direction to produce a data array in the horizontal 
direction, for each of said plurality of images; and 

extracting feature points in said reference image and 
feature points in each of the images except for said reference 
image based on the respective data arrays in the vertical and 
horizontal directions. 

14. The imaging method according to claim 9, comprising the step 
of: 

extracting feature points in said reference image and feature 
points in each of the images except for said reference image based 
on luminance information included in said plurality of images. 
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ABSTRAC T OF THE PI SCLOS RE 

An image capturing apparatus and a method therefor are 
proposed for the production of a natural composite image 
corresponding to an object to be imaged. A misregistration amount 
(d) of each image (xSl) except for a reference image (xLl) is 
detected with respect to the reference image, misregistration of 
each image except for the reference image with respect to the 
reference image is corrected on the basis of the misregistration 
amount, and then the reference image and each misregistration 
corrected image (xS4) except for the reference image are composed, 
thereby making it possible to avoid a degraded image quality which 
would otherwise be produced when a plurality of images are 
composed without correction, and consequently produce a natural 
composite image corresponding to an object to be imaged. 
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any PCT International application which designated at least 
one country other than the United States, listed below and 
have also identified below, by checking the box, any foreign 
application for patent or inventor's certificate, or PCT 
International application having a filing date before that of the 
application on which priority is claimed. 


Prior Foreign Application (s) 

P1 1-152808 Japan 
(Number) (Country) 


Priority Not Claimed 

31 Mav 1999 
(Day/Month/Year Filed) 

out* jib) 
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(Number) 



(Country) 



(Day/Month/Year Filed) 



fcli. Iff 3 5«*B&&1 1 9^fe(e)gCH:3lv^TTiew* 



I hereby claim the benefit under Title 35, United States Code, 
Section 119(e) of any United States provisional application(s) 
listed below. 



(Application No.) 



(Filing Date) 

WU B) 



(Application No.) 



(Filing Date) 



I hereby claim the benefit under Title 35, United States Code, 
Section 120 of any United States application(s), or 365(c) of 
any PCT International application designating the United 
States, listed below and, insofar as the subject matter of 
each of the claims of this application is not disclosed in the 
prior United States or PCT international application in the 
manner provided by the first paragraph of Title 35, United 
States Code, Section 112, I acknowledge the duty to disclose 
information which is material to patentability as defined in 
Title 37, Code of Federal Regulations, Section 1.56 which 
became available between the filing date of the prior 
application and the national or PCT International filing date of 
application. 



^Application No.) 



(Application No.) 



(Filing Date) 

(fcHHB) 



(Filing Date) 



(Status: Patented, Pending, Abandoned) 

out i »«Fff*i». 



(Status: Patented, Pending, Abandoned) 

mist - #HFfr-«r». 



tut. *a#©teattX-p^T*i***-e»#?ffc5* 

1 8S«1 0 0 l*K3£i**> t>L<!4* 



I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on 
information and belief are believed to be true; and further that 
these statements were made with the knowledge that willful 
false statements and the like so made are punishable by fine 
or imprisonment, or both, under Section 1001 of Title 18 of 
the United States Code and that such willful false statements 
may be jeopardize the validity of the application or any patent 
issued thereon. 
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tit, TE«tt«*v^ift. (#a±, *fc»fta 


POWER OF ATTORNEY: As a named inventor, 1 hereby 
appoint the following attorney(s) and/or agent(s) to prosecute 
this application and transact all business in the Patent and 
Trademark office connected therewith (list name and 
registration number) 


Karl A. Limbach 18,689 
George C. Limbach 19,305 
John K. Uiikema 20,282 
Neil A. Smith 25,441 
Veronica C. Devitt 29,375 
Ronald L. Yin 27,607 
Gerald T. Sekimura 30,103 
Michael A. Stallman 29,444 
Philip A. Girard 28,848 
Michael J. Poilock 29,098 


Steven M. Everett 30,050 
Alfred A Equitz 30,922 
Charles P. Sammut 28,901 
Mark C. Pickering 36,239 
Patricia Coleman James 37,155 
Kathleen A Frost 37,326 
Alan A Limbach 39,749 
Douglas C. Limbach 35,249 
Seong-Kun Oh* 

* Recognition under 37 CFR 10 9(b) 


Cameron A King 41,897 
Kyla L Harriet 41,815 
Mayumi Maeda 40,075 
Michael R. Ward 38,651 
Roger S. Sampson 44,314 
Charles L. Hamilton 42,624 
Andrew V. Smith 43,132 
Eric N. Hoover 37,355 
J. Thomas McCarthy 22,420 
Joel G. Ackerman 24,307 




^onrl f^orrp^nnnrlpnr , p tcv 

OKI l\J OUI 1 COfJUl IUCI 1 v^C IU. 

Charles P. Sammut, Esq. 
Limbach & Limbach L.L.P. 
2001 Ferry Building 
San Francisco, CA 94111-4262 




Direct Telephone Calls to: (name and telephone 
number) 

Charles P. Sammut 
(415) 433-4150 




Full name of sole or first inventor: 
MASAMI OGATA 




Inventor's signature Date 




Residence 

Kanagawa, Japan 


mn 


Citizenship 
Japan 




Post Office Address 

c/o SONY CORPORATION 
7-35, Kitashinagawa 6-chome 
Shinagawa-ku, Tokyo, 141-0001 JAPAN 



Page 3 of 4 



Revision of PTO/SB/1 06 (8-96) 



Japanese Language Declaration 




Full name of second joint inventor, if any 
TAKASHI TSUCHIYA 






Second inventor's signature 


Date 




Residence 

Tokyo, Japan 




Citizenship 
Japan 




Post Office Address 






c/o SONY CORPORATION 
7-35, Kitashinagawa 6-chome 
Shinagawa-ku, Tokyo, 141-0001 JAPAN 






Full name of third joint inventor, if any 






KAZUHIKO UEDA 




SlHMm* Btt 


Third inventor's signature 


Date 




Residence 

Kanagawa, Japan 




Citizenship 
Japan 


ass 


Post Office Address 

c/o SONY CORPORATION 
7-35, Kitashinagawa 6-chome 
Shinagawa-ku, Tokyo, 141-0001 JAPAN 
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